-
Notifications
You must be signed in to change notification settings - Fork 52
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Refactor] support more file/array schema in tools #292
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
在 APIHub 上面预支 tool 的执行结果: ============================= test session starts ==============================
platform darwin -- Python 3.10.13, pytest-7.4.2, pluggy-1.3.0 -- /Users/wujingjing05/miniconda3/envs/eb-agent/bin/python
cachedir: .pytest_cache
rootdir: /Users/wujingjing05/projects/yiyan/ERNIE-Bot-SDK-add-tool-unittest/erniebot-agent
plugins: cov-4.1.0, asyncio-0.23.2, anyio-4.2.0
asyncio: mode=strict
collecting ... collected 24 items
tests/integration_tests/apihub/test_car_cls.py::TestRemoteTool::test_car_classify PASSED [ 4%]
tests/integration_tests/apihub/test_dish.py::TestRemoteTool::test_dish_classify PASSED [ 8%]
tests/integration_tests/apihub/test_doc_rm_bnd.py::TestRemoteTool::test_doc_rm_bnd PASSED [ 12%]
tests/integration_tests/apihub/test_handwriting.py::TestRemoteTool::test_hand_text_rec PASSED [ 16%]
tests/integration_tests/apihub/test_high_ocr.py::TestRemoteTool::test_tool PASSED [ 20%]
tests/integration_tests/apihub/test_image_moderation.py::TestRemoteTool::test_tool PASSED [ 25%]
tests/integration_tests/apihub/test_img_enhance.py::TestRemoteTool::test_tool PASSED [ 29%]
tests/integration_tests/apihub/test_img_transform.py::TestRemoteTool::test_img_style_trans PASSED [ 33%]
tests/integration_tests/apihub/test_img_transform.py::TestRemoteTool::test_person_animation PASSED [ 37%]
tests/integration_tests/apihub/test_landmark.py::TestRemoteTool::test_dish_classify PASSED [ 41%]
tests/integration_tests/apihub/test_ocr.py::TestRemoteTool::test_formula FAILED [ 45%]
tests/integration_tests/apihub/test_ocr.py::TestRemoteTool::test_ocr_general PASSED [ 50%]
tests/integration_tests/apihub/test_ocr.py::TestRemoteTool::test_ocr_pp PASSED [ 54%]
tests/integration_tests/apihub/test_ocr.py::TestRemoteTool::test_pp_ocr_v4 PASSED [ 58%]
tests/integration_tests/apihub/test_ocr.py::TestRemoteTool::test_shopping_receipt FAILED [ 62%]
tests/integration_tests/apihub/test_pic_translate.py::TestRemoteTool::test_tool FAILED [ 66%]
tests/integration_tests/apihub/test_pp_models.py::TestPPRemoteTool::test_pp_matting PASSED [ 70%]
tests/integration_tests/apihub/test_pp_models.py::TestPPRemoteTool::test_pp_ocr_v4 PASSED [ 75%]
tests/integration_tests/apihub/test_pp_models.py::TestPPRemoteTool::test_pp_structure PASSED [ 79%]
tests/integration_tests/apihub/test_pp_models.py::TestPPRemoteTool::test_pp_tinypose PASSED [ 83%]
tests/integration_tests/apihub/test_pp_models.py::TestPPRemoteTool::test_pp_vehicle PASSED [ 87%]
tests/integration_tests/apihub/test_text_moderation.py::TestRemoteTool::test_tool PASSED [ 91%]
tests/integration_tests/apihub/test_text_to_speech.py::TestRemoteTool::test_text_to_speech PASSED [ 95%]
tests/integration_tests/apihub/test_translation.py::TestRemoteTool::test_tool FAILED [100%]
=================================== FAILURES ===================================
_________________________ TestRemoteTool.test_formula __________________________
self = <tests.integration_tests.apihub.test_ocr.TestRemoteTool testMethod=test_formula>
@pytest.mark.asyncio
async def test_formula(self):
> toolkit = RemoteToolkit.from_aistudio("formula", file_manager=self.file_manager)
tests/integration_tests/apihub/test_ocr.py:69:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/erniebot_agent/tools/remote_toolkit.py:261: in from_aistudio
return cls.from_url(tool_url, version=version, access_token=access_token, file_manager=file_manager)
src/erniebot_agent/tools/remote_toolkit.py:305: in from_url
toolkit = RemoteToolkit.from_openapi_dict(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'erniebot_agent.tools.remote_toolkit.RemoteToolkit'>
openapi_dict = {'errorCode': 500, 'errorMsg': '工具不存在', 'logId': '8b8c797e20df8dc01527aff20169f34c', 'servers': [{'url': 'https://tool-formula.aistudio-hub.baidu.com'}]}
access_token = '4ce50e3378f418d271c480c8ddfa818537071dbe'
file_manager = <erniebot_agent.file.file_manager.FileManager object at 0x106cc2470>
@classmethod
def from_openapi_dict(
cls,
openapi_dict: Dict[str, Any],
access_token: Optional[str] = None,
file_manager: Optional[FileManager] = None,
) -> RemoteToolkit:
> info = EndpointInfo(**openapi_dict["info"])
E KeyError: 'info'
src/erniebot_agent/tools/remote_toolkit.py:175: KeyError
------------------------------ Captured log call -------------------------------
WARNING asyncio:base_events.py:1904 Executing <Task pending name='Task-71' coro=<IsolatedAsyncioTestCase._asyncioLoopRunner() running at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:95> wait_for=<Future pending cb=[Task.task_wakeup()] created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/asyncio/base_events.py:429> created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:117> took 0.467 seconds
WARNING asyncio:base_events.py:1904 Executing <Task pending name='Task-71' coro=<IsolatedAsyncioTestCase._asyncioLoopRunner() running at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:95> wait_for=<Future pending cb=[Task.task_wakeup()] created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/asyncio/base_events.py:429> created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:117> took 0.228 seconds
_____________________ TestRemoteTool.test_shopping_receipt _____________________
self = <tests.integration_tests.apihub.test_ocr.TestRemoteTool testMethod=test_shopping_receipt>
@pytest.mark.asyncio
async def test_shopping_receipt(self):
> toolkit = RemoteToolkit.from_aistudio("shopping-receipt", file_manager=self.file_manager)
tests/integration_tests/apihub/test_ocr.py:58:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/erniebot_agent/tools/remote_toolkit.py:261: in from_aistudio
return cls.from_url(tool_url, version=version, access_token=access_token, file_manager=file_manager)
src/erniebot_agent/tools/remote_toolkit.py:305: in from_url
toolkit = RemoteToolkit.from_openapi_dict(
src/erniebot_agent/tools/remote_toolkit.py:185: in from_openapi_dict
parameter_view = ToolParameterView.from_openapi_dict(schema)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'erniebot_agent.tools.schema.ToolParameterView'>
schema = {'properties': {'log_id': {'type': 'integer'}, 'pdf_file_size': {'type': 'string'}, 'words_result': {'properties': {'c...d', 'pdf_file_size', 'words_result'], 'type': 'object', 'x-apifox-orders': ['log_id', 'pdf_file_size', 'words_result']}
@classmethod
def from_openapi_dict(cls, schema: dict) -> Type[ToolParameterView]:
"""parse openapi component schemas to ParameterView
Args:
response_or_returns (dict): the content of status code
Returns:
_type_: _description_
"""
# TODO(wj-Mcat): to load Optional field
fields = {}
for field_name, field_dict in schema.get("properties", {}).items():
# skip loading invalid field to improve compatibility
if "type" not in field_dict:
raise ToolError(f"`type` field not found in `{field_name}` property", stage="Loading")
if "description" not in field_dict:
> raise ToolError(f"`description` field not found in `{field_name}` property", stage="Loading")
E erniebot_agent.utils.exceptions.ToolError: An error occured in stage <Loading>. The error message is `description` field not found in `log_id` property
src/erniebot_agent/tools/schema.py:250: ToolError
------------------------------ Captured log call -------------------------------
WARNING asyncio:base_events.py:1904 Executing <Task pending name='Task-95' coro=<IsolatedAsyncioTestCase._asyncioLoopRunner() running at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:95> wait_for=<Future pending cb=[Task.task_wakeup()] created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/asyncio/base_events.py:429> created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:117> took 1.080 seconds
WARNING asyncio:base_events.py:1904 Executing <Task pending name='Task-95' coro=<IsolatedAsyncioTestCase._asyncioLoopRunner() running at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:95> wait_for=<Future pending cb=[Task.task_wakeup()] created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/asyncio/base_events.py:429> created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:117> took 0.310 seconds
___________________________ TestRemoteTool.test_tool ___________________________
self = <tests.integration_tests.apihub.test_pic_translate.TestRemoteTool testMethod=test_tool>
@pytest.mark.asyncio
async def test_tool(self):
toolkit = RemoteToolkit.from_aistudio("pic-translate", file_manager=self.file_manager)
file = await self.file_manager.create_file_from_path(self.download_fixture_file("shouxiezi.png"))
agent = self.get_agent(toolkit)
> result = await agent.run("这张照片里面讲了啥?", files=[file])
tests/integration_tests/apihub/test_pic_translate.py:18:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/erniebot_agent/agents/agent.py:129: in run
raise e
src/erniebot_agent/agents/agent.py:126: in run
agent_resp = await self._run(prompt, files)
src/erniebot_agent/agents/function_agent.py:149: in _run
curr_step, new_messages = await self._step(chat_history)
src/erniebot_agent/agents/function_agent.py:190: in _step
tool_resp = await self.run_tool(tool_name=tool_name, tool_args=tool_args)
src/erniebot_agent/agents/agent.py:176: in run_tool
raise e
src/erniebot_agent/agents/agent.py:173: in run_tool
tool_resp = await self._run_tool(tool, tool_args)
src/erniebot_agent/agents/agent.py:251: in _run_tool
tool_ret = await tool(**parsed_tool_args)
src/erniebot_agent/tools/remote_tool.py:139: in __call__
tool_arguments = await self.__pre_process__(tool_arguments)
src/erniebot_agent/tools/remote_tool.py:117: in __pre_process__
tool_arguments = self.tool_view.parameters(**tool_arguments).model_dump(mode="json")
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = OpenAPIParameterView(image=b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x03\x06\x00\x00\x03\xf8\x08\x02\x00\x00\x00\x...\xd8W\xe2\x8b\xe6k7c\xaeN+\x03.\xd7\xfe?eP\xf75\xb6\xa4\x95L\x00\x00\x00\x00IEND\xaeB`\x82', from='auto', to='zh', v=3)
def model_dump(
self,
*,
mode: Literal['json', 'python'] | str = 'python',
include: IncEx = None,
exclude: IncEx = None,
by_alias: bool = False,
exclude_unset: bool = False,
exclude_defaults: bool = False,
exclude_none: bool = False,
round_trip: bool = False,
warnings: bool = True,
) -> dict[str, Any]:
"""Usage docs: https://docs.pydantic.dev/2.5/concepts/serialization/#modelmodel_dump
Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
Args:
mode: The mode in which `to_python` should run.
If mode is 'json', the dictionary will only contain JSON serializable types.
If mode is 'python', the dictionary may contain any Python objects.
include: A list of fields to include in the output.
exclude: A list of fields to exclude from the output.
by_alias: Whether to use the field's alias in the dictionary key if defined.
exclude_unset: Whether to exclude fields that have not been explicitly set.
exclude_defaults: Whether to exclude fields that are set to their default value from the output.
exclude_none: Whether to exclude fields that have a value of `None` from the output.
round_trip: Whether to enable serialization and deserialization round-trip support.
warnings: Whether to log warnings when invalid fields are encountered.
Returns:
A dictionary representation of the model.
"""
> return self.__pydantic_serializer__.to_python(
self,
mode=mode,
by_alias=by_alias,
include=include,
exclude=exclude,
exclude_unset=exclude_unset,
exclude_defaults=exclude_defaults,
exclude_none=exclude_none,
round_trip=round_trip,
warnings=warnings,
)
E UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid utf-8
../../../../miniconda3/envs/eb-agent/lib/python3.10/site-packages/pydantic/main.py:308: UnicodeDecodeError
----------------------------- Captured stderr call -----------------------------
�[92mINFO - [Run][Start] FunctionAgent is about to start running with input:
�[94m这张照片里面讲了啥?�[92m�[0m
�[92mINFO - [LLM][Start] ERNIEBot is about to start running with input:
role: �[94muser�[92m
content: �[94m这张照片里面讲了啥?
<file>file-local-da8582b4-b3a6-11ee-803b-eac7af941eda</file>�[92m �[0m
�[92mINFO - [LLM][End] ERNIEBot finished running with output:
role: �[93massistant�[92m
function_call: �[93m
{
"name": "pic-translate/v1.0/pic_translate",
"thoughts": "用户想要知道图片中的文字内容,我需要调用OCR工具来识别图片中的文字,然后根据需要翻译成中文。任务拆解:[sub-task1: 使用OCR工具识别图片中的文字,sub-task2: 根据需要将文字翻译成中文]。接下来,我需要调用[pic-translate/v1.0/pic_translate]来翻译图片中的文字。",
"arguments": "{\"image\":\"file-local-da8582b4-b3a6-11ee-803b-eac7af941eda\",\"to\":\"zh\"}"
}�[92m �[0m
�[92mINFO - [Tool][Start] �[95mRemoteTool�[92m is about to start running with input:
�[95m{
"image": "file-local-da8582b4-b3a6-11ee-803b-eac7af941eda",
"to": "zh"
}�[92m�[0m
------------------------------ Captured log call -------------------------------
WARNING asyncio:base_events.py:1904 Executing <Task pending name='Task-98' coro=<IsolatedAsyncioTestCase._asyncioLoopRunner() running at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:101> wait_for=<Future pending cb=[shield.<locals>._outer_done_callback() at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/asyncio/tasks.py:864, Task.task_wakeup()] created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/asyncio/base_events.py:429> created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:117> took 1.090 seconds
___________________________ TestRemoteTool.test_tool ___________________________
self = <tests.integration_tests.apihub.test_translation.TestRemoteTool testMethod=test_tool>
@pytest.mark.asyncio
async def test_tool(self):
> toolkit = RemoteToolkit.from_aistudio("translation", file_manager=self.file_manager)
tests/integration_tests/apihub/test_translation.py:13:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/erniebot_agent/tools/remote_toolkit.py:261: in from_aistudio
return cls.from_url(tool_url, version=version, access_token=access_token, file_manager=file_manager)
src/erniebot_agent/tools/remote_toolkit.py:305: in from_url
toolkit = RemoteToolkit.from_openapi_dict(
src/erniebot_agent/tools/remote_toolkit.py:185: in from_openapi_dict
parameter_view = ToolParameterView.from_openapi_dict(schema)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'erniebot_agent.tools.schema.ToolParameterView'>
schema = {'properties': {'log_id': {'description': 'logId', 'type': 'integer'}, 'result': {'properties': {'from': {'description... '翻译结果数组列表', 'items': {'properties': {...}, 'type': 'object'}, 'type': 'array'}}, 'type': 'object'}}, 'type': 'object'}
@classmethod
def from_openapi_dict(cls, schema: dict) -> Type[ToolParameterView]:
"""parse openapi component schemas to ParameterView
Args:
response_or_returns (dict): the content of status code
Returns:
_type_: _description_
"""
# TODO(wj-Mcat): to load Optional field
fields = {}
for field_name, field_dict in schema.get("properties", {}).items():
# skip loading invalid field to improve compatibility
if "type" not in field_dict:
raise ToolError(f"`type` field not found in `{field_name}` property", stage="Loading")
if "description" not in field_dict:
> raise ToolError(f"`description` field not found in `{field_name}` property", stage="Loading")
E erniebot_agent.utils.exceptions.ToolError: An error occured in stage <Loading>. The error message is `description` field not found in `result` property
src/erniebot_agent/tools/schema.py:250: ToolError
------------------------------ Captured log call -------------------------------
WARNING asyncio:base_events.py:1904 Executing <Task pending name='Task-152' coro=<IsolatedAsyncioTestCase._asyncioLoopRunner() running at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:95> wait_for=<Future pending cb=[Task.task_wakeup()] created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/asyncio/base_events.py:429> created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:117> took 0.271 seconds
=========================== short test summary info ============================
FAILED tests/integration_tests/apihub/test_ocr.py::TestRemoteTool::test_formula
FAILED tests/integration_tests/apihub/test_ocr.py::TestRemoteTool::test_shopping_receipt
FAILED tests/integration_tests/apihub/test_pic_translate.py::TestRemoteTool::test_tool
FAILED tests/integration_tests/apihub/test_translation.py::TestRemoteTool::test_tool
=================== 4 failed, 20 passed in 237.27s (0:03:57) ===================
|
以上报错的几个 tool 是由于 yaml 内容错误导致的。 也就是现在的 PR 能够兼容之前tool 的文件解析格式。 |
Bobholamovic
approved these changes
Jan 16, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
File部分LGTM
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.